When the Best Move Isn't Optimal: Q-learning with Exploration

The most popular delayed reinforcement learning technique, Q-learning (Watkins 1989)) estimates the future reward expected from executing each action in every state. If these estimates are correct, then an agent can use them to select the action with maximal expected future reward in each state, and thus perform optimally. Watkins has proved that Q-learning produces an optimal policy (the function mapping states to actions) and that these estimates converge to the correct values given the optimal policy. However, often the agent does not follow the optimal policy faithfully the agent must also explore the world, taking suboptimal actions in order to learn more about its environment. The “optimal” policy produced by Q-learning is no longer optimal if its prescriptions are only followed occasionally. In many situations (e.g., dynamic environments), the agent never stops exploring. In such domains Q-learning converges to policies which are suboptimal in the sense that there exists a different policy which would achieve higher reward when combined with exploration. A bit of notation: &(z, a) is the expected future reward received after taking action a in state x. V(x) is the expected future reward received after starting in state x. 0 and v are used to denote the approximations kept by the algorithm. Each time the agent takes an action a moving it from state x to state y and generating a reward T, Q-learning updates the approximations according to the following rules: